Railway - Status

Recent notices

Completed
September 19, 2024 at 9:08 PM
Completed
September 19, 2024 at 9:08 PM
We have successfully upgraded our edge network hardware. You should not experience any further connection disconnects.
Our monitors during this time show no degradation to TCP connections or HTTP requests failures while we performed this across our fleet but as mentioned, closing your connection to shutdown the machine was unavoidable. Your clients would reconnect and continue to work as expected.
Update
September 19, 2024 at 8:17 PM
In progress
September 19, 2024 at 8:17 PM
We are upgrading our final set of machines ahead of schedule.
In progress
September 19, 2024 at 6:43 PM
In progress
September 19, 2024 at 6:43 PM
This upgrade is in progress.
Planned
September 19, 2024 at 6:00 PM
Planned
September 19, 2024 at 6:00 PM
We are upgrading the hardware for our edge network that powers TCP and HTTP proxying. During this time requests to boxes being upgraded will have 5 minutes to complete before being disconnected. These clients will be able to reconnect and will be routed to another box that will handle the traffic indefinitely.

Resolved
September 19, 2024 at 10:42 AM
Resolved
September 19, 2024 at 10:42 AM
Queue back to zero
Monitoring
September 19, 2024 at 10:34 AM
Monitoring
September 19, 2024 at 10:34 AM
Queued backpressure is almost drained. Most queued deploys have been rolled out
Identified
September 19, 2024 at 10:28 AM
Identified
September 19, 2024 at 10:28 AM
Deploys are going out and the queue is draining. We will move to monitoring when there are no queued builds
Investigating
September 19, 2024 at 10:13 AM
Investigating
September 19, 2024 at 10:13 AM
Builds are currently in a queueing state. We're lookinig into it

Resolved
September 18, 2024 at 5:46 PM
Resolved
September 18, 2024 at 5:46 PM
We identified a caching issue on the dashboard where high latency on domain creation caused the cached domains list to become stale. The new domains were created and successfully propagated to proxies, but failed to appear in the UI. We've rectified the caching issue and reduced the upstream timeouts to the external vendors API. This is incident is now resolved.
Identified
September 18, 2024 at 4:40 PM
Identified
September 18, 2024 at 4:40 PM
Due to an ongoing incident with an upstream provider since approximately 1pm UTC, custom and service domain creation can take upto 15s. During this time, the dashboard state of domains may go out of sync. We're lowering the timeout to our upstream provider to remediate this.

Resolved
September 16, 2024 at 10:26 AM
Resolved
September 16, 2024 at 10:26 AM
This incident has been resolved. Our Europe West edge network was hit by a L4 DoS attack between 10:46AM UTC 10:52AM UTC. Our mitigations were successfully applied. Users connecting from Europe may have noticed increased latency during this period and a small number of dropped requests.
Identified
September 16, 2024 at 10:05 AM
Identified
September 16, 2024 at 10:05 AM
We have identified the issue and are seeing full recovery across Europe.
Investigating
September 16, 2024 at 9:55 AM
Investigating
September 16, 2024 at 9:55 AM
Users may encounter issues connecting to Railway-hosted workloads from Europe. We are currently investigating this incident.

Between 7am and 8am UTC, staged changes failed to apply for some deployments. We have deployed a fix.

Resolved
September 14, 2024 at 7:00 AM
Resolved
September 14, 2024 at 7:00 AM
(This is a retroactive incident) We experienced a partial degradation in our US West edge network (V2 proxy) during 06:43AM UTC to 06:47AM UTC. Railway-hosted workloads may be inaccessible to users connecting from US West during this period. This was the result of a erroneous code deploy. It paged us immediately and the change was reverted within a few minutes. Service has been fully restored since 06:47AM UTC.